Papers with syllogistic reasoning
Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning (2023.findings-acl)
Copied to clipboard
Yongkang Wu, Meng Han, Yutao Zhu, Lei Li, Xinyu Zhang, Ruofei Lai, Xiaoguang Li, Yuanhang Ren, Zhicheng Dou, Zhao Cao
| Challenge: | SylloBase is a benchmark for syllogistic reasoning, a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. |
| Approach: | They propose to use a benchmark to learn syllogistic reasoning on a set of templates and to use them to generate and understand slogisms. |
| Outcome: | The proposed benchmark covers a complete taxonomy of syllogism reasoning patterns, and contains both automatically and manually constructed samples. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
Teaching Small Language Models to Learn Logic through Meta-Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly evaluated on reasoning tasks, yet their logical abilities remain contested. |
| Approach: | They propose to apply few-shot meta-learning to large language models' reasoning domain to enable them to acquire abstract inference patterns that generalize to novel structures. |
| Outcome: | The proposed model outperforms GPT-4o and o3-mini on a syllogistic reasoning task. |
A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)
Copied to clipboard
| Challenge: | syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences. |
| Approach: | They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results. |
| Outcome: | The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning. |
Immediate Inference: The Missing Foundation in Large Language Model Logical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work on LLMs has focused on fine-grained skill decomposition and consistency probing at the propositional level. |
| Approach: | They propose a benchmark evaluating immediate inference that evaluates elemental operations over categorical propositions and proposes a model that uses immediate inferential reasoning. |
| Outcome: | The proposed benchmark demonstrates that models lack robust operator grounding, oscillating between structural reasoning and surface pattern matching, inconsistent handling of quantifiers and negation. |
Syllogistic Reasoning for Legal Judgment Analysis (2023.emnlp-main)
Copied to clipboard
Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, Pengjie Ren
| Challenge: | Legal judgment assistants are developing fast due to impressive progress of large language models. |
| Approach: | They construct and manually correct a syllogistic reasoning dataset for legal judgment analysis using large language models as benchmarks. |
| Outcome: | The proposed dataset contains 11,239 criminal cases covering 4 criminal elements, 80 charges and 124 articles. |
Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models of large language reasoning exhibit reasoning biases similar to humans, a study shows . syllogistic reasoning is one of the basic forms of deductive reasoning . |
| Approach: | They propose to use a syllogism dataset to evaluate models' reasoning abilities . they propose to ask LLMs to translate slogismatic slurs into abstract logical expressions . |
| Outcome: | The proposed method shows that models exhibit reasoning biases similar to humans, and that there is room for improvement in reasoning problems where premises and hypotheses are neither entailment nor contradiction. |
Thinking in Schemas: Robust Syllogistic Reasoning in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | syllogistic reasoning models often mistake what sounds true for what is formally valid . content effect is a limitation of sluggish reasoning, which can lead to invalid conclusions . eisape et al., 2024: a key open problem for formal inference in natural language. |
| Approach: | They propose a schema-guided framework that disentangles semantic plausibility from logical validity. |
| Outcome: | The proposed framework outperforms existing frameworks while reducing CE. |